The impact of a smartphone‐based cognitive aid on clinical performance during cardiac arrest simulations: A randomized controlled trial

Abstract Objectives In‐hospital cardiac arrests are common and associated with high mortality. Smartphone applications offer quick access to algorithms and timers but often lack real‐time guidance. This study assesses the impact of the Code Blue Leader application on the performance of providers leading cardiac arrest simulations. Methods This open‐label randomized controlled trial included Advanced Cardiac Life Support (ACLS)–trained medical doctors (MD) and registered nurses (RN). Participants were randomized to lead the same ACLS simulation with or without the app. The primary outcome, “performance score,” was assessed by a trained rater using a validated ACLS scoring system. Secondary outcomes included percentage of critical actions performed, number of incorrect actions, and chest compression fraction (percentage of time spent performing chest compressions). A sample size of 30 participants was calculated to detect a difference of 20% at the 0.05 alpha level with 90% power. Results Fifteen MDs and 15 RNs underwent stratified randomization. The median (interquartile range) performance score in the app group was 95.3% (93.0%–100.0%) compared to 81.4% (60.5%–88.4%) in the control group, demonstrating an effect size of r = 0.69 (Z = −3.78, r = 0.69, p = 0.0002). The percentage of critical actions performed in the app group was 100% (96.2%–100.0%) compared to 85.0% (74.1%–92.4%) in the control group. The number of incorrect actions performed in the app group was 1 (1) compared to 4 (3–5) in the control group. Chest compression fraction in the app group was 75.5% (73.0%–84.0%) compared to 75.0% (72.0%–85.0%) in the control group. Conclusions The Code Blue Leader smartphone app significantly improved the performance of ACLS‐trained providers in cardiac arrest simulations.


INTRODUC TI ON Background
The incidence of adult in-hospital cardiac arrests (IHCA) in the United States is 10.16 per 1000 hospital admissions, which is approximately 292,000 per year. 1,2 Only 26.7% of these patients survive until discharge. 2 The likelihood of return of spontaneous circulation (ROSC) from IHCA and survival to discharge is positively correlated with adherence to American Heart Association's Advanced Cardiovascular Life Support (ACLS) guidelines. [3][4][5] More specifically, factors that are associated with decreased likelihood of ROSC or survival to discharge include delays in cardiopulmonary resuscitation (CPR) initiation, greater time to first defibrillation, incorrect voltage or timing of defibrillation, delays in rhythm identification, delays in medication administration or incorrect medications as per ACLS guidelines, and greater CPR interruptions, including those associated with endotracheal intubation. [3][4][5] IHCA outcomes are also impacted by policies, including those that regulate resuscitation team design, mock codes, nursing empowerment, and frequency of IHCA case review. 6,7 Team design involves establishing resuscitation teams composed of highly trained health care professionals (HCPs) with defined responsibilities whose resuscitation role supersedes other clinical duties.
This facilitates an organized response to IHCAs where all team members are familiar with each other and their roles. 6 Nursing empowerment involves providing education and modifying policies to permit nurses to initiate interventions independently, such as rapid defibrillation and ACLS medication administration. 6 While policy reform can improve IHCA outcomes to a degree, 6 even experienced HCPs struggle to lead teams when cognitive overload occurs. 8,9 Therefore, other interventions may help to address this threat to patient care. 9

Importance
HCPs who lead IHCAs are often ACLS-trained; however, the stress of these critical situations can exceed the cognitive capacity required to effectively follow ACLS algorithms. 9 The Yerkes-Dodson law describes this phenomenon with their bell-shaped model of arousal; while moderate levels of arousal yield optimal performance, higher levels are associated with increased cognitive errors and a decline in performance. 9 Recognition of the negative impact of higher levels of stress during IHCAs has led to the development of cognitive aids that reduce cognitive load and improve adherence to the American Heart Association's guidelines. Using cognitive aids during IHCAs has been found to decrease stress, improve teamwork, and lead to calmer work environments. 10 This enabled HCPs to focus more on clinical decision making and patient status, which was associated with better overall patient care. Cognitive aids used during IHCAs may include timers, metronomes, algorithm cards, and smartphone applications. 1, [11][12][13][14] There are many smartphone apps available that offer access to algorithm cards, timers, and metronomes. Unfortunately, very few offer real-time guidance and audible prompts. Metelmann et al. 13 found that out of 34 basic life support apps reviewed, only five provided step-by-step guidance with accurate American Heart Association information and only one app received an above-average score on the System Usability Scale (SUS).

Goals of this investigation
We sought to assess the utility of the Code Blue Leader smartphone application. This app offers both accurate and up-to-date ACLS guidelines, while also providing real-time, step-by-step guidance to reduce cognitive load and support HCPs leading IHCAs. It is critically important to evaluate all new cognitive aids in simulated environments as this offers the opportunity to assess safety and efficacy and identify areas of improvement before use in high-stakes clinical settings. 9 Therefore, the purpose of this study is to assess the impact of the Code Blue Leader smartphone application on the performance of ACLS-trained HCPs leading cardiac arrest simulations.

Study design and setting
This study received approval through a harmonized ethical review between the Island Health Research Ethics Board and the University of Victoria Research Ethics Board. The smartphone application under investigation was created and is owned by the principal investigator (PI) of this study. As such, the PI and other affiliated author were not present in the simulation laboratory during any participant involvement and did not contribute to any data collection or participant assessment. This conflict of interest was disclosed in the application for ethical review. This was an open-label, randomized controlled trial involving ACLS-certified medical doctors (MD) and registered nurses (RN). The trial was conducted in the high-fidelity simulation laboratory situated in the Centre for Interprofessional Clinical Simulation Learning in the Royal Jubilee Hospital in Victoria, BC. A total of seven trial sessions were held from February to April 2021. Because the intervention evaluated was not implemented in a clinical setting, no patients were involved and no health-related or clinical outcomes were assessed on human participants. Therefore, this study was considered a nonclinical randomized controlled trial and therefore was not preregistered. This trial was not funded or sponsored by any company or institution. No grants or other methods of financial support were utilized.

Study population
Eligible participants included MDs, RNs, nurse practitioners, licensed practical nurses, advanced care paramedics, and students of these professions who were working, training, or affiliated with the Royal Jubilee Hospital or Victoria General Hospital and held an active ACLS certificate within 2 years of trial participation. The participants were recruited via posters, handout materials, and email recruitment. Only MDs and RNs responded to recruitment efforts.
Recruitment was on a first-come basis until an equal distribution (50% MDs, 50% RNs) was achieved.

Study protocol
The order of participation was based on order of recruitment.
Written, informed consent was obtained from all participants. Prior to randomization, each participant underwent a 10-min teaching session on how to use the Code Blue Leader smartphone application (version 0.1) based on a standardized checklist of learning points. The Code Blue Leader smartphone application provides real-time, stepby-step guidance and audible prompts for the 2020 ACLS cardiac arrest algorithms. This application was developed for both Apple iOS and Android and is currently undergoing beta testing to prepare for release on both platforms. The Code Blue Leader application will be available to the general population in the near future; however, at the time of writing the manuscript, it was not yet available for download.
Participants were then randomized via random-number generator to lead the same high-fidelity cardiac arrest simulation, either with or without the app. They were stratified based on professional qualification (MD vs. RN) to ensure equal proportions in each study arm ( Figure 1). The participants could not be blinded to arm allocation due to the nature of the study. Participants randomized to the app group were given an unlocked smartphone with the Code Blue Leader application preinstalled and were encouraged to use the smartphone app during the ACLS scenario. Those randomized to the control group were told to use any cognitive aids they would normally use to lead the ACLS scenario, including algorithm cards, timers, other smartphone applications, etc.
The simulation lab equipment was set up identically prior to each simulation, based on a standardized checklist. All participants ran the same ACLS scenario in which they responded to a "Code Blue" on an unknown hospital ward. The simulated patient went into cardiac arrest immediately upon participant arrival and sequentially progressed through four pulseless rhythms: ventricular tachycardia, ventricular fibrillation, pulseless electrical activity, and asystole.
During the simulation, each participant would lead the scenario independently. They were accompanied by three medically trained assistants in the room who would respond to prompts or follow orders from the participant but would not prompt them or perform any actions independently. Only one trial participant was involved during an individual simulation.

Outcomes
The primary outcome was the performance score, calculated as the percentage of total correct actions performed. Secondary outcomes included the percentage of critical outcomes performed, the number of incorrect actions, and the chest compression fraction, calculated as the percentage of time spent performing chest compressions. The primary outcome as well as two of the secondary outcomes, percentage of critical outcomes performed and the number of incorrect actions, were assessed based on data collected by a rater using a validated ACLS scoring system. The scoring system used is available as supplemental material accompanying the online article ( Figure S1). 15 The scoring system detailed both correct and incorrect actions to be performed in each ACLS algorithm tested and delineated which of these actions were considered "critical." As this checklist was validated prior to the most recent ACLS guidelines, a single criterion (the use of atropine) in the "correct actions" category was removed to stay up to date with the current guidelines. The rater was ACLS certified, trained how to use this checklist, and blinded to the study outcomes.
Inter-rater reliability was assessed between the trial rater and an external rater for five practice scenarios prior to the start of the trial, yielding a kappa coefficient of 0.95, which is considered an excellent inter-rater reliability. A SimMan 3G simulator was used to run the preprogrammed scenario described above and to collect objective data for the final secondary outcome, chest compression fraction.
Following the simulation, to determine usability of the smartphone app, a validated SUS 16 was filled out by participants randomized to the app group. The overall SUS score is calculated by converting each question's score to a new number, adding them together, then multiplying by 2.5 to convert the original scores of 0-40 to 0-100.

Data analysis
A target sample size of 30 participants was calculated to detect an effect size of 20% at the 0.05 alpha level with 90% power. The likely F I G U R E 1 Participant progress through study protocol.
performance of the control group (mean ± SD performance score of 66.2% ± 10.4%) was estimated using data from the validation study for the ACLS checklist used in this trial. 15 Results were analyzed using STATA 15.1. Shapiro-Wilk test for normality, which indicated a nonnormal distribution of the primary outcome data for the app group. Therefore, continuous data are described using median (interquartile range) and comparison for outcomes between groups was accomplished using the Wilcoxon rank-sum test adhering to intention-to-treat analysis.

Characteristics of study participants
A total of 15 MDs and 15 RNs were included, and all participants completed the simulation. Between three and five participants ran through the simulation per session. The app group and the control group had similar baseline characteristics (Table 1). Among the MDs, a variety of specialties were represented including family medicine, internal medicine, emergency medicine, anesthesia, radiology, and radiation oncology.

Main results
There was a significant difference in the primary outcome (performance scores) between the two groups ( Table 2 and Figure 2). The performance score in the app group was 95.3% (93.0%-100.0%) compared to 81.4% (60.5%-88.4%) in the control group, with the Wilcoxon rank-sum test demonstrating a significant estimated effect size of r = 0.69 (Z = −3.78, r = 0.69, p = 0.0002). Prespecified subgroup analysis based on qualification also demonstrates a difference between the intervention and control groups in the primary outcome.
For the secondary outcomes, Table 3 demonstrates a significant difference between the two groups for the median percentage of critical actions performed, which was 100% (96.2%-100.0%) in the app group compared to 85.0% (74.1%-92.4%) in the control group.
The median number of incorrect actions performed was significant with the app group at 1 (1) compared to 4 (3)(4)(5) in the control group.
The median percentage of time spent performing chest compressions was similar between groups at 75.5% (73.0%-84.0%) in the app group compared to 75.0% (72.0%-85.0%) in the control group.
The cognitive aids used by the control group included timers, metronomes, and smartphone resources with ACLS algorithms.
Unfortunately, the number of participants using each aid and the characteristics of smartphone resources was not collected in further detail.  (Table S1).

DISCUSS ION
Use of the Code Blue Leader app significantly improved ACLS performance scores among ACLS-certified HCPs. This impact was demonstrated among prespecified subgroups (MDs and RNs).
In comparison to the app group, the control group performed significantly less "critical" correct actions while leading cardiac arrest simulations despite still having access to their usual cognitive aids of choice (i.e., timers, metronomes, ACLS algorithms, other smartphone applications). While many cognitive aids have been demonstrated to improve performance when compared to no cognitive aids at all, it is also important to compare different modalities to explore the most efficacious approach. For example, Burden et al. 17 found that simply providing algorithm cards is not as beneficial as real-time guidance and prompts. In their study, one group received algorithm cards, while the other group assigned a medical student to read the cards aloud. In the group without the "reader," 66% of the participants read the cognitive aid silently and then quickly placed it back on the cart; 33% of these participants then picked it up again, briefly read it, and again placed it back on the cart. While reading the algorithm card, participants stopped engaging with the team entirely.
The group with an assigned reader performed significantly more critical actions demonstrating the importance of step-by-step guidance.
Assigning a reader is not always logistically feasible in situations with TA B L E 1 Baseline characteristics for study participants.

Smartphone App No App
Age ( Abbreviations: SUS, System Usability Score. a SUS score > 68 is considered above-average usability.

TA B L E 4 SUS for participants
randomized to use the smart phone app scored from 1 (strongly disagree) to 5 (strongly agree).
limited human resources so smartphone applications, such as Code Blue Leader, are a reasonable alternative to fulfill this role.
The control group performed significantly more incorrect actions when compared to the app group. These incorrect actions included, but were not limited to, not identifying the correct cardiac rhythm on the monitor, and administering medications that are not recommended for routine use during ACLS. Spanos and Patterson 18 discovered that a major barrier for effectively leading an IHCA is accurately identifying the rhythm on the monitor. In their study, half of medical residents identified ventricular fibrillation incorrectly, which led to a significant increase in time to defibrillation. The Code Blue Leader app includes schematics of each rhythm that participants were required to select to designate the ACLS algorithm they were following. This feature was included to support HCPs who lead IHCA infrequently and may benefit from a real-time visual reminder of each rhythm. Additionally, Benz 3 reports that using medications inappropriately during IHCAs not only contributes to critical shortages of lifesaving medications but is also associated with decreased rates of ROSC. It should be noted that in the primary characteristics of the study groups, the mean time since completing ACLS certification was less in the app group than the control group (10 months vs. 12 months). It is possible that the difference in recency of the certification may be one of the factors contributing to the differences in correct and incorrect actions between the two groups.
The compression fraction is an important outcome to assess as it is known to be a critical prognostic factor during a resuscitation and pre-

CON CLUS IONS
In summary, the Code Blue Leader smartphone app significantly improved the performance of Advanced Cardiovascular Life Support-trained health care professionals in a high-fidelity cardiac arrest simulation. Use of the app also improved the percentage of critical actions performed, decreased the number of incorrect actions performed, and did not affect the chest compression fraction.
Overall, the app was found by participants to have above-average usability. Those who contributed to the project but did not write the manuscript are listed in the acknowledgements section.

ACK N OWLED G M ENTS
This trial would not have been possible without the consistent and

CO N FLI C T O F I NTER E S T S TATEM ENT
This trial was not funded or sponsored by any company or institution. No grants or other methods of financial support were utilized.
The smartphone application under investigation in this trial was created by the principal investigator (SB) of this study. It is owned by SB and TJ and is currently protected under a provisional patent. As such, SB and TJ were not present in the simulation laboratory during any participant involvement and did not contribute to any data collection or participant assessment. This conflict of interest was disclosed in the application for ethical review.